Interpreting Human Gesture with Computer Vision

This paper discusses the use of computer vision for interpreting human gesture. We first present the functional roles of human gesture and observe that in real life, gesture may be enhanced with physical instruments. In Human Computer Interaction, input devices such as the data-glove, are convenient tricks for computer scientists to sense human gesture but are not necessarily ecological instruments. We claim that computer vision can turn the human (the whole as well as the parts) into an effective input device without intrusive add-on’s. In the last sections of the paper, we present FingerPaint, our demonstration of the use of cross-correlation as a means of tracking natural pointing devices (e.g., bare fingers, pens and erasers) in the context of the digital desk. 1. The Functional Roles of Human Gesture Human gesture serves three functional roles [Cadoz 94]: semiotic, ergotic, and epistemic. • The semiotic function of gesture is to communicate meaningful information. The structure of a semiotic gesture is conventional and commonly results from shared cultural experience. The goodbye gesture, the American sign language, the operational gestures used to guide airplanes on the ground, and even the vulgar “finger”, each illustrates the semiotic function of gesture. • The ergotic function of gesture is associated with the notion of work. It corresponds to the capacity of humans to manipulate the real world, to create artefacts, or to change the state of the environment by “direct manipulation”. Shaping pottery from clay, wiping dust, etc. result from ergotic gestures. • The epistemic function of gesture allows humans to learn from the environment through tactile experience. By moving your hand over an object, you appreciate its structure, you may discover the material it is made of, and so on. All three functions may be augmented using an instrument: a handkerchief for the semiotic good-bye gesture, a turn-table for the ergotic shape-up gesture of pottery, or a dedicated artefact to explore the world (for example, a retro-active system such as the pantograph [Ramstein 94] to sense the invisible). 2. Gesture in HCI: Computer Input Devices are “tricky instruments” In Human Computer Interaction, gesture has been primarily exploited for its ergotic function: typing on a keyboard, moving a mouse, and clicking buttons. The epistemic role of gesture has emerged effectively